Electronic device for communicating in augmented reality and method thereof

ABSTRACT

An augmented reality (AR) device supporting an AR is provided. The AR device includes a display, a communication circuit, at least one processor operatively connected to the display and the communication circuit, and a memory operatively connected to the at least one processor. The memory stores instructions that, when executed, cause the at least one processor to establish a connection with a user device storing a contact application and a message application, through the communication circuit, detect that the message application is executed, and display a first graphic user interface (GUI) and at least one avatar, which is disposed at a location adjacent to the first GUI and corresponds to at least one contact associated with the contact application or the message application, through the display in the AR.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(e) of a U.S. Provisional application Ser. No. 63/008,618, filed onApr. 10, 2020, in the U.S. Patent and Trademark Office, and under 35U.S.C. § 119(a) of a Korean patent application number 10-2020-0063058,filed on May 26, 2020, in the Korean Intellectual Property Office, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a technology for communication in augmentedreality (AR). More particularly, the disclosure relates to an AR deviceincluding a display, a communication circuit, and at least one processoroperatively connected to the display and the communication circuit, anda memory operatively connected to the at least one processor.

2. Description of Related Art

As an AR device, such as a wearable device or a head mounted display(HMD) is developed technologically, a user may experience AR. Forexample, the AR device may display a user interface (UI), such as anicon or an execution screen of an application in a real world to whichthe user belongs.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

To experience an augmented reality (AR), a user needs to wear an ARdevice. Even while wearing the AR device, the user needs to launch anapplication of a user device, such as a smartphone. For example, theuser needs to communicate with family, friends, or acquaintances bylaunching a message application installed in the user device while theuser wears the AR device.

Therefore, a need exists for an AR device including a display, acommunication circuit, and at least one processor operatively connectedto the display and the communication circuit, and a memory operativelyconnected to the at least one processor.

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea technology for communication in AR.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, an AR device supportingan AR is provided. The AR device includes a display, a communicationcircuit, at least one processor operatively connected to the display andthe communication circuit, and a memory operatively connected to the atleast one processor. The memory may store instructions that, whenexecuted, cause the at least one processor to establish a connectionwith a user device storing a contact application and a messageapplication, through the communication circuit, to detect that themessage application is executed, and to display a first graphic userinterface (GUI) and at least one avatar, which is disposed at a locationadjacent to the first GUI and corresponds to at least one contactassociated with the contact application or the message application,through the display in the AR.

In accordance with another aspect of the disclosure, an AR systemsupporting an AR is provided. The AR system includes an AR device and auser device. The AR device may include a first display and a firstcommunication circuit. The user device may include a second display, asecond communication circuit, at least one processor operativelyconnected to the second display and the second communication circuit,and a memory operatively connected to the at least one processor. Thememory may store at least one contact associated with a messageapplication or a contact application. The memory may store instructionsthat, when executed, cause the at least one processor to establish aconnection with the AR device through the second communication circuit,to detect that the message application is executed, to render a firstGUI and at least one avatar, which are to be displayed in the AR throughthe first display, as a 3 dimensional (3D) image, and to transmit therendered first GUI and the rendered at least one avatar to the AR devicethrough the second communication circuit. The at least one avatar maycorrespond to the at least one contact.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates a graphic user interface (GUI) and an avatar, whichare displayed on an augmented reality according to an embodiment of thedisclosure;

FIG. 2 illustrates a block diagram of an electronic device in a networkenvironment according to an embodiment of the disclosure;

FIG. 3 illustrates a block diagram of a user device according to anembodiment of the disclosure;

FIG. 4 is a block diagram of a program module according to an embodimentof the disclosure;

FIG. 5 is a block diagram of an augmented reality (AR) device accordingto an embodiment of the disclosure;

FIG. 6 is a diagram of a message application combined with a hardwareconfiguration of an AR system according to an embodiment of thedisclosure;

FIG. 7 illustrates an operation flowchart of an AR system for displayinga first GUI and an avatar according to an embodiment of the disclosure;

FIG. 8A illustrates a first GUI including an avatar according to anembodiment of the disclosure;

FIG. 8B illustrates an avatar corresponding to a contact according to anembodiment of the disclosure;

FIG. 9 illustrates a graphic effect of moving an avatar from a first GUIto a second GUI in an AR according to an embodiment of the disclosure;

FIG. 10A illustrates an operation of detecting a gesture for an avataraccording to an embodiment of the disclosure;

FIG. 10B illustrates a graphic effect indicating a response of an avatarto a gesture according to an embodiment of the disclosure;

FIG. 11 is an operation flowchart of an AR system for providing agraphic effect in which an avatar moves according to an embodiment ofthe disclosure;

FIG. 12 illustrates an operation of changing a message thread accordingto an embodiment of the disclosure;

FIG. 13A illustrates a user interface (UI) for entering a messageaccording to an embodiment of the disclosure;

FIG. 13B illustrates a UI for entering a message according to anembodiment of the disclosure;

FIG. 14 illustrates a UI for displaying a message according to anembodiment of the disclosure;

FIG. 15A illustrates an operation of outputting a message depending on aviewing area according to an embodiment of the disclosure;

FIG. 15B illustrates an operation flowchart of an AR device foroutputting a message depending on a viewing area according to anembodiment of the disclosure;

FIG. 16 illustrates a UI for displaying a received message according toan embodiment of the disclosure;

FIG. 17 illustrates a second GUI including a plurality of avatars in anAR according to an embodiment of the disclosure;

FIG. 18 illustrates an operation of selecting one of a plurality ofavatars disposed on a second GUI according to an embodiment of thedisclosure;

FIG. 19 illustrates an operation of sharing content in an AR accordingto an embodiment of the disclosure;

FIG. 20 illustrates an interaction scenario in a second GUI according toan embodiment of the disclosure; and

FIG. 21 illustrates an image including an avatar according to anembodiment of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary sill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 1 illustrates a GUI and an avatar, which are displayed in an ARaccording to an embodiment of the disclosure.

Referring to FIG. 1, in a real space 10, an AR device 101 may be worn ona part (e.g., a head) of a body of a user 50. The AR device 101 mayrefer to a device including a display capable of supporting an AR. Forexample, the AR device 101 may include at least one of a wearable deviceand a head mounted display (HMD).

According to an embodiment of the disclosure, the AR device 101 mayoutput a first GUI 30 for displaying a contact stored in a messageapplication or a contact application of a user device (e.g., asmartphone, not illustrated) in 3 dimensional (3D) in an AR 20. Forexample, the first GUI 30 may be referred to as a ‘nest’.

According to an embodiment of the disclosure, the first GUI 30 may befixed in a specified area of the real space 10. For example, when a userinput for placing the first GUI 30 on a table 12 of the real space 10 isreceived, the AR device 101 may display the first GUI 30 on the table12. In this case, even though the AR device 101 moves by a movement ofthe user 50, a location of the first GUI 30 may not be changed in thereal space 10. According to another embodiment of the disclosure, thefirst GUI 30 may be fixed to a specified location of an area (e.g., aviewing area 1510 of FIG. 15A) that the user 50 is capable of viewingthrough the AR device 101. In this case, when the AR device 101 moves bythe movement of the user 50, the first GUI 30 may move together with themovement of the AR device 101.

The first GUI 30 may include at least one avatar (e.g., 32, 34, or 36)corresponding to each contact stored in the user device. In thisspecification, it may be described that the first GUI 30 ‘includes’ anavatar. However, according to various embodiments of the disclosure, theavatars 32, 34, and 36 in the AR 20 may be positioned at variouslocations adjacent to the first GUI 30. For example, referring to anenlarged view 22 in which the AR 20 is enlarged, the avatars 32, 34, and36 may be disposed on the first GUI 30. As another example, although notillustrated in FIG. 1, the avatars 32, 34, and 36 may be disposed below,side, or inside the first GUI 30.

The avatars 32, 34, and 36 may represent contacts stored in the userdevice, respectively. According to embodiments of the disclosure, a userof the user device (or the AR device 101) may select one of templateavatars that have been stored in advance, and thus an appearance of theavatars 32, 34, and 36 may be determined, an appearance of the avatars32, 34, and 36 may be arbitrarily determined based on information (e.g.,an age, a gender, or a country) included in contact information.Alternatively, an appearance of the avatars 32, 34, and 36 may bedetermined based on settings of a counterpart.

A size of the AR space 20 capable of being provided by the AR device 101is restricted as compared to the number of contacts stored in the userdevice, and thus the AR device 101 according to an embodiment maydisplay only an avatar corresponding to at least one pre-selectedcontact among contacts stored in the user device. For example, the ARdevice 101 may display, on the first GUI 30, a contact (e.g.,“favorite”) selected by user settings or a contact having a highfrequency of communication.

FIG. 2 illustrates a block diagram illustrating an electronic device ina network environment 200 according to an embodiment of the disclosure.

Referring to FIG. 2, an electronic device 201 in the network environment200 may communicate with an external electronic device 202 via a firstnetwork 298 (e.g., a short-range wireless communication network), or atleast one of an external electronic device 204 or a server 208 via asecond network 299 (e.g., a long-range wireless communication network).According to an embodiment of the disclosure, the electronic device 201may communicate with the external electronic device 204 via the server208. According to an embodiment of the disclosure, the electronic device201 may include a processor 220, memory 230, an input module 250, asound output module 255, a display module 260, an audio module 270, asensor module 276, an interface 277, a connecting terminal 278, a hapticmodule 279, a camera module 280, a power management module 288, abattery 289, a communication module 290, a subscriber identificationmodule (SIM) 296, or an antenna module 297. In an embodiment of thedisclosure, at least one of the components (e.g., the connectingterminal 278) may be omitted from the electronic device 201, or one ormore other components may be added in the electronic device 201. In anembodiment of the disclosure, some of the components (e.g., the sensormodule 276, the camera module 280, or the antenna module 297) may beimplemented as a single component (e.g., the display module 260).

The processor 220 may execute, for example, software (e.g., a program240) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 201 coupled with theprocessor 220, and may perform various data processing or computation.According to one embodiment of the disclosure, as at least part of thedata processing or computation, the processor 220 may store a command ordata received from another component (e.g., the sensor module 276 or thecommunication module 290) in a volatile memory 232, process the commandor the data stored in the volatile memory 232, and store resulting datain a non-volatile memory 234. According to an embodiment of thedisclosure, the processor 220 may include a main processor 221 (e.g., acentral processing unit (CPU) or an application processor (AP)), or anauxiliary processor 223 (e.g., a graphics processing unit (GPU), aneural processing unit (NPU), an image signal processor (ISP), a sensorhub processor, or a communication processor (CP)) that is operableindependently from, or in conjunction with, the main processor 221. Forexample, when the electronic device 201 includes the main processor 221and the auxiliary processor 223, the auxiliary processor 223 may beadapted to consume less power than the main processor 221, or to bespecific to a specified function. The auxiliary processor 223 may beimplemented as separate from, or as part of the main processor 221.

The auxiliary processor 223 may control at least some of functions orstates related to at least one component (e.g., the display module 260,the sensor module 276, or the communication module 290) among thecomponents of the electronic device 201, instead of the main processor221 while the main processor 221 is in an inactive (e.g., a sleep)state, or together with the main processor 221 while the main processor221 is in an active state (e.g., executing an application). According toan embodiment of the disclosure, the auxiliary processor 223 (e.g., animage signal processor or a communication processor) may be implementedas part of another component (e.g., the camera module 280 or thecommunication module 290) functionally related to the auxiliaryprocessor 223. According to an embodiment of the disclosure, theauxiliary processor 223 (e.g., the neural processing unit) may include ahardware structure specified for artificial intelligence modelprocessing. An artificial intelligence model may be generated by machinelearning. Such learning may be performed, e.g., by the electronic device201 where the artificial intelligence is performed or via a separateserver (e.g., the server 208). Learning algorithms may include, but arenot limited to, e.g., supervised learning, unsupervised learning,semi-supervised learning, or reinforcement learning. The artificialintelligence model may include a plurality of artificial neural networklayers. The artificial neural network may be a deep neural network(DNN), a convolutional neural network (CNN), a recurrent neural network(RNN), a restricted boltzmann machine (RBM), a deep belief network(DBN), a bidirectional recurrent deep neural network (BRDNN), deepQ-network or a combination of two or more thereof but is not limitedthereto. The artificial intelligence model may, additionally oralternatively, include a software structure other than the hardwarestructure.

The memory 230 may store various data used by at least one component(e.g., the processor 220 or the sensor module 276) of the electronicdevice 201. The various data may include, for example, software (e.g.,the program 240) and input data or output data for a command relatedthererto. The memory 230 may include the volatile memory 232 or thenon-volatile memory 234.

The program 240 may be stored in the memory 230 as software, and mayinclude, for example, an operating system (OS) 242, middleware 244, oran application 246.

The input module 250 may receive a command or data to be used by anothercomponent (e.g., the processor 220) of the electronic device 201, fromthe outside (e.g., a user) of the electronic device 201. The inputmodule 250 may include, for example, a microphone, a mouse, a keyboard,a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 255 may output sound signals to the outside ofthe electronic device 201. The sound output module 255 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record. The receiver maybe used for receiving incoming calls. According to an embodiment of thedisclosure, the receiver may be implemented as separate from, or as partof the speaker.

The display module 260 may visually provide information to the outside(e.g., a user) of the electronic device 201. The display module 260 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment of thedisclosure, the display module 260 may include a touch sensor adapted todetect a touch, or a pressure sensor adapted to measure the intensity offorce incurred by the touch.

The audio module 270 may convert a sound into an electrical signal andvice versa. According to an embodiment of the disclosure, the audiomodule 270 may obtain the sound via the input module 250, or output thesound via the sound output module 255 or a headphone of an externalelectronic device (e.g., the external electronic device 202) directly(e.g., wiredly) or wirelessly coupled with the electronic device 201.

The sensor module 276 may detect an operational state (e.g., power ortemperature) of the electronic device 201 or an environmental state(e.g., a state of a user) external to the electronic device 201, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment of the disclosure, the sensormodule 276 may include, for example, a gesture sensor, a gyro sensor, anatmospheric pressure sensor, a magnetic sensor, an acceleration sensor,a grip sensor, a proximity sensor, a color sensor, an infrared (IR)sensor, a biometric sensor, a temperature sensor, a humidity sensor, oran illuminance sensor.

The interface 277 may support one or more specified protocols to be usedfor the electronic device 201 to be coupled with the external electronicdevice (e.g., the external electronic device 202) directly (e.g.,wiredly) or wirelessly. According to an embodiment of the disclosure,the interface 277 may include, for example, a high definition multimediainterface (HDMI), a universal serial bus (USB) interface, a securedigital (SD) card interface, or an audio interface.

A connecting terminal 278 may include a connector via which theelectronic device 201 may be physically connected with the externalelectronic device (e.g., the external electronic device 202). Accordingto an embodiment of the disclosure, the connecting terminal 278 mayinclude, for example, a HDMI connector, a USB connector, a SD cardconnector, or an audio connector (e.g., a headphone connector).

The haptic module 279 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment of the disclosure, the hapticmodule 279 may include, for example, a motor, a piezoelectric element,or an electric stimulator.

The camera module 280 may capture a still image or moving images.According to an embodiment of the disclosure, the camera module 280 mayinclude one or more lenses, image sensors, image signal processors, orflashes.

The power management module 288 may manage power supplied to theelectronic device 201. According to one embodiment of the disclosure,the power management module 288 may be implemented as at least part of,for example, a power management integrated circuit (PMIC).

The battery 289 may supply power to at least one component of theelectronic device 201. According to an embodiment of the disclosure, thebattery 289 may include, for example, a primary cell which is notrechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module 290 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 201 and the external electronic device (e.g., theexternal electronic device 202, the external electronic device 204, orthe server 208) and performing communication via the establishedcommunication channel. The communication module 290 may include one ormore communication processors that are operable independently from theprocessor 220 (e.g., the application processor (AP)) and supports adirect (e.g., wired) communication or a wireless communication.According to an embodiment of the disclosure, the communication module290 may include a wireless communication module 292 (e.g., a cellularcommunication module, a short-range wireless communication module, or aglobal navigation satellite system (GNSS) communication module) or awired communication module 294 (e.g., a local area network (LAN)communication module or a power line communication (PLC) module). Acorresponding one of these communication modules may communicate withthe external electronic device via the first network 298 (e.g., ashort-range communication network, such as Bluetooth™, wireless-fidelity(Wi-Fi) direct, or infrared data association (IrDA)) or the secondnetwork 299 (e.g., a long-range communication network, such as a legacycellular network, a 5G network, a next-generation communication network,the Internet, or a computer network (e.g., LAN or wide area network(WAN)). These various types of communication modules may be implementedas a single component (e.g., a single chip), or may be implemented asmulti components (e.g., multi chips) separate from each other. Thewireless communication module 292 may identify and authenticate theelectronic device 201 in a communication network, such as the firstnetwork 298 or the second network 299, using subscriber information(e.g., international mobile subscriber identity (IMSI)) stored in thesubscriber identification module 296.

The wireless communication module 292 may support a 5G network, after a4G network, and next-generation communication technology, e.g., newradio (NR) access technology. The NR access technology may supportenhanced mobile broadband (eMBB), massive machine type communications(mMTC), or ultra-reliable and low-latency communications (URLLC). Thewireless communication module 292 may support a high-frequency band(e.g., the mmWave band) to achieve, e.g., a high data transmission rate.The wireless communication module 292 may support various technologiesfor securing performance on a high-frequency band, such as, e.g.,beamforming, massive multiple-input and multiple-output (massive MIMO),full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, orlarge scale antenna. The wireless communication module 292 may supportvarious requirements specified in the electronic device 201, an externalelectronic device (e.g., the external electronic device 204), or anetwork system (e.g., the second network 299). According to anembodiment of the disclosure, the wireless communication module 292 maysupport a peak data rate (e.g., 20 Gbps or more) for implementing eMBB,loss coverage (e.g., 164 dB or less) for implementing mMTC, or U-planelatency (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL),or a round trip of 1 ms or less) for implementing URLLC.

The antenna module 297 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 201. According to an embodiment of the disclosure, theantenna module 297 may include an antenna including a radiating elementincluding a conductive material or a conductive pattern formed in or ona substrate (e.g., a printed circuit board (PCB)). According to anembodiment of the disclosure, the antenna module 297 may include aplurality of antennas (e.g., array antennas). In such a case, at leastone antenna appropriate for a communication scheme used in thecommunication network, such as the first network 298 or the secondnetwork 299, may be selected, for example, by the communication module290 (e.g., the wireless communication module 292) from the plurality ofantennas. The signal or the power may then be transmitted or receivedbetween the communication module 290 and the external electronic devicevia the selected at least one antenna. According to an embodiment of thedisclosure, another component (e.g., a radio frequency integratedcircuit (RFIC)) other than the radiating element may be additionallyformed as part of the antenna module 297.

According to various embodiments of the disclosure, the antenna module297 may form an mmWave antenna module. According to an embodiment of thedisclosure, the mmWave antenna module may include a printed circuitboard, a RFIC disposed on a first surface (e.g., the bottom surface) ofthe printed circuit board, or adjacent to the first surface and capableof supporting a designated high-frequency band (e.g., the mmWave band),and a plurality of antennas (e.g., array antennas) disposed on a secondsurface (e.g., the top or a side surface) of the printed circuit board,or adjacent to the second surface and capable of transmitting orreceiving signals of the designated high-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment of the disclosure, commands or data may betransmitted or received between the electronic device 201 and theexternal electronic device 204 via the server 208 coupled with thesecond network 299. Each of the external electronic devices 202 or 204may be a device of a same type as, or a different type, from theelectronic device 201. According to an embodiment of the disclosure, allor some of operations to be executed at the electronic device 201 may beexecuted at one or more of the external electronic devices 202, 204, or208. For example, if the electronic device 201 should perform a functionor a service automatically, or in response to a request from a user oranother device, the electronic device 201, instead of, or in additionto, executing the function or the service, may request the one or moreexternal electronic devices to perform at least part of the function orthe service. The one or more external electronic devices receiving therequest may perform the at least part of the function or the servicerequested, or an additional function or an additional service related tothe request, and transfer an outcome of the performing to the electronicdevice 201. The electronic device 201 may provide the outcome, with orwithout further processing of the outcome, as at least part of a replyto the request. To that end, a cloud computing, distributed computing,mobile edge computing (MEC), or client-server computing technology maybe used, for example. The electronic device 201 may provideultra-low-latency services using, e.g., distributed computing or mobileedge computing. In another embodiment of the disclosure, the externalelectronic device 204 may include an internet-of-things (IoT) device.The server 208 may be an intelligent server using machine learningand/or a neural network. According to an embodiment of the disclosure,the external electronic device 204 or the server 208 may be included inthe second network 299. The electronic device 201 may be applied tointelligent services (e.g., smart home, smart city, smart car, orhealthcare) based on 5G communication technology or IoT-relatedtechnology.

The electronic device 201 illustrated in FIG. 2 may be the AR device 101of FIG. 1 or a device (e.g., a user device) that is electricallyconnected to the AR device 101 and stores contacts.

FIG. 3 illustrates a block diagram of a user device according to anembodiment of the disclosure.

Referring to FIG. 3, a user device 301 may refer to a device configuredto communicate with a counterpart stored in a message application or acontact application. For example, the user device 301 may include asmartphone, a tablet personal computer (PC), a mobile phone, a videophone, an e-book reader, a desktop PC, a laptop computer, a netbookcomputer, a workstation, a personal digital assistant (PDA), a portablemultimedia player (PMP), a moving picture experts group phase 1 or phase2 (MPEG-1 or MPEG-2) audio layer 3 (MP3) player, a mobile medicaldevice, or a home appliance.

An AR device 302 may be implemented as a wearable device (e.g., a HMD)capable of being worn by a user. The AR device 302 may include a displayfor providing an AR when the user device 301 operates in an AR mode. Forexample, when the user device 301 is mounted on the AR device 302, theuser device 301 may execute the AR mode. As another example, when theuser device 301 is not mounted on the AR device 302, the user device 301may execute the AR mode depending on user settings or execution of anapplication associated with the AR mode. When the AR mode is executed,the user device 301 may three-dimensionally render a GUI (e.g., thefirst GUI 30 and the avatars 32, 34, and 36 of FIG. 1) displayed in theAR through the AR device 302. In this case, two screens corresponding tothe user's eyes (e.g., a left eye and a right eye) may be output througha display of the AR device 302.

The user device 301 may include a touch screen 320, a controller 330, amemory 340, and a communication circuit 350. As another example, theuser device 301 may further include at least one of other componentsincluded in the electronic device 201 of FIG. 2.

The touch screen 320 may perform a function that is identical or similarto the function of the display module 260 of FIG. 2. The touch screen320 may include a display panel 321 and/or a touch panel 322. Each ofthe display panel 321 and the touch panel 322 may be implemented as aseparate panel. As another example, the touch screen 320 may includeonly the display panel 321 without including the touch panel 322.

The controller 330 may perform a function that is identical or similarto the function of the processor 220 of FIG. 2. The controller 330 mayinclude an AR mode processing module 331, an event detection module 332,an event information processing module 333, and an applicationcontroller 334. Modules included in the controller 330 may meanconfigurations implemented by hardware or software. When implemented insoftware, the modules included in the controller 330 may be stored inthe memory 340 as an instruction set. The controller 330 may performoperations of the user device 301 by executing the instruction set.

When the user device 301 operates in the AR mode, the AR mode processingmodule 331 may process operations associated with the AR mode. Forexample, the AR mode processing module 331 may load at least one ARprogram 341 stored in the memory 340.

The event detection module 332 may detect an event that occurs when theAR mode is operated. The event detection module 332 may determinewhether there is information to be displayed on a screen associated withan event occurring when the AR mode is operated. For example, whencontent, such as a message is received from a counterpart's electronicdevice through a message application, the event detection module 332 maydetect that an event occurs.

The event information processing module 333 may process information thatis to be detected by the event detection module 332 and is to bedisplayed in an AR. For example, the event information processing module333 may convert information (e.g., a text message) associated with adetected event into a 3D image to be displayed in the AR. In this case,the event information processing module 333 may convert two-dimensional(2D) information into information corresponding to left and right eyes,may synthesize the converted information, and may render the synthesizedinformation to be displayed on the AR device 302.

The application controller 334 may control an application when the userdevice 301 operates in the AR mode. For example, when a messageapplication is executed in the AR mode, the application controller 334may block other applications or operate other applications in thebackground such that other applications do not interfere with executionof the message application.

The memory 340 may perform a function that is identical or similar tothe function of the memory 230 of FIG. 2. The memory 340 may store theAR program 341 and event-related information 342. The AR program 341 maybe an application associated with an AR mode operation of the userdevice 301. The event-related information 342 may include informationassociated with an event occurring in the AR.

The communication circuit 350 may perform a function that is identicalor similar to the function of the communication module 190 of FIG. 2.The communication circuit 350 may be used to be electrically connectedto the AR device 302 or an external server. For example, thecommunication circuit 350 may be connected to the AR device 302 throughBluetooth, Wi-Fi, or tethering. As another example, the communicationcircuit 350 may include a wired interface.

FIG. 4 is a block diagram of a program module according to an embodimentof the disclosure.

Referring to FIG. 4, a program module 400 may include a configurationthat is identical or similar to that of the program 240 of FIG. 2. Theprogram module 400 may include a system operating system (e.g., anoperating system (OS)) 410, a framework 420, and an application 430.

The system operating system 410 may perform a function that is identicalor similar to the function of the operating system 242 of FIG. 2. Thesystem operating system 410 may include at least one system resourcemanager or at least one device driver. The system resource manager maycontrol, allocate, or recover a system resource. The system resourcemanager may include at least one manager, such as a process manager, amemory manager, or a file system manager. The device driver may includeat least one driver, such as a display driver, a camera driver, aBluetooth driver, a shared memory driver, an USB driver, a keypaddriver, a Wi-Fi driver, an audio driver, or an inter-processcommunication (IPC) driver.

For example, the framework 420 may include the middleware 244 of FIG. 2.The framework 420 may provide a function necessary for an applicationthrough an application programming interface (API) such that theapplication is capable of efficiently using limited system resources inan electronic device (e.g., the user device 301 in FIG. 3). When theelectronic device operates in an AR mode, the framework 420 may controla function associated with the AR mode. For example, the framework 420may control at least one AR application (e.g., a plurality of ARapplications 451).

The application 430 may perform a function that is identical or similarto the function of the application 246 of FIG. 2. The application 430may include a plurality of AR applications (e.g., the plurality of ARapplications 451) and a plurality of general applications (e.g., 452).The AR application may refer to an application that operates in the ARmode, and, the general application may refer to an application thatoperates in a normal mode, not the AR mode.

According to an embodiment of the disclosure, the application 430 mayfurther include an AR control application 440. At least one ARapplication and/or at least one general application 450 may becontrolled by the AR control application 440. For example, the ARcontrol application 440 may perform a function that is identical orsimilar to the function of at least one of the AR mode processing module331, the event detection module 332, the event information processingmodule 333, or the application controller 334 of FIG. 3.

According to an embodiment of the disclosure, when a user deviceoperates in the AR mode or the normal mode, the system operating system410 may notify the framework 420 that an event occurs (e.g., receiving atext message from a counterpart's electronic device). When an eventoccurs in the case where the user device operates in the normal mode,the framework 420 may control execution of the general application suchthat a screen associated with the event is displayed through a display(e.g., the touch screen 320 in FIG. 3) of the user device. When an eventoccurs in the case where the user device operates in the AR mode, theframework 420 may control the general application such that a screenassociated with the event is displayed only through an AR device (e.g.,the AR device 302 in FIG. 3) and is not displayed through a display ofthe user device, thereby preventing unnecessary resource consumption andpreventing the general application from interfering with an operation inthe AR mode. According to another embodiment of the disclosure, when anevent occurs in the case where the user device operates in the AR mode,the framework 420 may provide an environment in which a user is capableof controlling an AR device through the user device, by controlling anevent-related screen to be displayed through the display of the userdevice as well as the AR device.

FIG. 5 is a block diagram of an AR device according to an embodiment ofthe disclosure.

Referring to FIG. 5, an AR device 501 may include a configuration thatis identical or similar to that of the AR device 101 of FIG. 1 or the ARdevice 302 of FIG. 3, and may perform a function that is identical orsimilar to that of the AR device 101 of FIG. 1 or the AR device 302 ofFIG. 3. The AR device 501 may include a memory 520, an input/outputinterface 530, a communication interface 540, a sensor 550, a display560, and a processor 570 that are operatively connected through a bus510.

The memory 520 may store a spatial map 521. For example, the spatial map521 may refer to spatial information about a real space (e.g., the realspace 10 of FIG. 1) surrounding the AR device 501. For example, when theAR device 501 is worn by a user, the AR device 501 may generate a spacesurrounding the AR device 501 as a 3D map by scanning a surrounding areathrough the sensor 550. The AR device 501 may identify a location of theAR device 501 by using the generated spatial map 521 or may identify alocation where a specific GUI (e.g., the first GUI 30 of FIG. 1 or theavatars 32, 34, and 36) is positioned.

The memory 520 may include an API 522 for providing a necessary functionto applications stored in the AR device 501.

The input/output interface 530 may include a hardware or softwareconfiguration for interfacing with a user. For example, the input/outputinterface 530 may include a microphone or a speaker.

The communication interface 540 may be configured to be connected to auser device (e.g., the user device 301 in FIG. 3) by wire or wirelessly.The communication interface 540 may also be referred to as a“communication module” or a “communication circuit”.

The sensor 550 may include at least one sensor. For example, the sensor550 may include at least one of a hand pose sensor 551 capable ofsensing a movement of the user's hand (or a finger), a head pose sensor552 capable of sensing a movement of the user's head, or another sensor553. For example, the other sensor 553 may include at least one of alocation measurement sensor capable of identifying a location of the ARdevice 501 in a real space, such as a global positioning system (GPS), amotion sensor capable of sensing a direction or movement of the ARdevice 501, such as an acceleration sensor, a geomagnetic sensor, and agyro sensor, a proximity sensor capable of sensing an external objectpositioned at a location close to the AR device 501, a red green blue(RGB) sensor capable of sensing an intensity of light or color aroundthe AR device 501, an illuminance sensor, or a hall sensor.

The display 560 may include a see-through display 565 to provide an ARto a user.

The processor 570 may be operatively connected to the memory 520, theinput/output interface 530, the communication interface 540, the sensor550, and the display 560, and may perform overall operations of the ARdevice 501.

FIG. 6 is a diagram of a message application 630 combined with ahardware configuration of an AR system according to an embodiment of thedisclosure.

Referring to FIG. 6, an AR device 610 and a user device 620 may beelectrically connected to each other. For example, the user device 620may be mounted in the AR device 610, the AR device 610 and the userdevice 620 may be connected through a wired interface. Alternatively,the AR device 610 and the user device 620 may be connected via wirelesscommunication. A processor 612 and a memory 614 of the AR device 610 mayperform functions that are identical or similar to functions of theprocessor 570 and the memory 520 of FIG. 5, respectively. A processor622 and a memory 624 of the user device 620 may perform functions thatare identical or similar to functions of the controller 330 and thememory 340 of FIG. 3, respectively.

The message application 630 may be stored in the memory 624 of the userdevice 620. In this case, the message application 630 may performfunctions that are identical or similar to those of the modules includedin the controller 330 of FIG. 3 or the AR control application 440 ofFIG. 4. For example, the message application 630 may render at least oneavatar 632 (e.g., the avatar 32, 34, or 36 of FIG. 1) and a first GUI634 (e.g., the first GUI 30 of FIG. 1) that are to be displayed througha see-through display 640 (e.g., the display 560 of FIG. 5) of the ARdevice 610, and may transmit the rendered first GUI 634 and the renderedat least one avatar 632 to the AR device 610. When a user input 635 toselect one of at least one avatar included in the first GUI 634 isreceived, the message application 630 may render a second GUI 638 and aselected avatar 636 such that the selected avatar 636 is displayed onthe second GUI 638 (e.g., a second GUI 920 in FIG. 9) indicating amessage thread. The message application 630 may transmit the renderedsecond GUI 638 and the selected avatar 636 to the AR device 610. Whilethe second GUI 638 is displayed, the message application 630 may receivea user input 650 to enter content. For example, the content may includeat least one of a text, an image, 3D content, or a voice.

The AR device 610 may output the first GUI 634, at least one avatar 632,the selected avatar 636, and the second GUI 638, which are renderedthrough the message application 630, through the see-through display640. According to an embodiment of the disclosure, the AR device 610 mayobtain the user input 635 to select an avatar and the user input 650 toenter content, through a sensor (e.g., the sensor 550 in FIG. 5).

FIG. 7 illustrates an operation flowchart 700 of an AR system fordisplaying a first GUI and an avatar according to an embodiment of thedisclosure. An AR system may refer to a system including an AR device(e.g., the AR device 610 of FIG. 6) and a user device (e.g., the userdevice 620 of FIG. 6).

Referring to FIG. 7, in operation 710, the AR system may establish aconnection between the AR device and the user device. The connectionbetween the AR device and the user device may be established by wire orwirelessly. For example, when the AR device is implemented with an HMD,the user device may be mounted in the AR device. As another example, theAR device and the user device may be tethered through a wired interfaceor a wireless communication protocol (e.g., Bluetooth or Wi-Fi).

In operation 720, the AR system may detect that a message application ofthe user device is executed while the connection between the AR deviceand the user device is established. For example, the message applicationmay include an application capable of communicating with another userdevice. For example, the message application may be replaced with a chatapplication, a video call application, a mail application, or a socialnetwork service (SNS) application.

In operation 730, the AR system may display a first GUI (e.g., GUI 30 ofFIG. 1) including at least one avatar (e.g., the avatar 32, 34, or 36 ofFIG. 1) in an AR. The first GUI may refer to an object in which a spacein which at least one avatar is positioned is visually displayed in theAR. The at least one avatar may be based on at least one contact storedin the executed message application or a contact application linked tothe message application.

FIG. 8A illustrates a first GUI including an avatar according to anembodiment of the disclosure.

Referring to FIG. 8A, an AR device may display a first GUI 820 in an AR.A shape of the first GUI 820 is not limited to an example shown in FIG.8A, and, a shape of the first GUI 820 may be changed by user settings.The first GUI 820 may be fixed to a specified location in a real spaceor may be fixed to a specified area of a display of an AR device.According to an embodiment of the disclosure, the AR device may displayat least one avatar 801, 802, or 803 at a location (e.g., a top surfaceof the first GUI) adjacent to the first GUI 820. The AR device maydisplay the specified number of avatars on the first GUI 820 based on asize of the first GUI 820. For example, the AR device may display, onthe first GUI 820, a contact, which is specified as a favorite dependingon user settings, or a contact having a high frequency of contact.

According to an embodiment of the disclosure, the AR device may displayicons 822, 824, and 826 of a menu for controlling the first GUI 820 orthe at least one avatar 801, 802, or 803 at a location adjacent to thefirst GUI 820. For example, the menu corresponding to the icons 822,824, and 826 may include at least one of editing of the first GUI 820,favorite settings, editing of an avatar, checking of recentcommunication, or editing of a contact.

FIG. 8B illustrates an avatar corresponding to a contact according to anembodiment of the disclosure.

Referring to FIG. 8B, an AR device may edit at least one avatar 801,802, or 803 or contact information 811, 812, or 813 respectivelycorresponding to the at least one avatar 801, 802, or 803. For example,when an avatar of a contact stored in a user device has not beengenerated yet, the AR device may apply one of the pre-stored templateavatars to a contact by a user's selection. When a counterpart of acontact, to which a template avatar is applied, generates a new avatar,an avatar stored in the user device may be changed from the templateavatar to an avatar generated by the counterpart.

FIG. 9 illustrates a graphic effect of moving an avatar from a first GUIto a second GUI in an AR according to an embodiment of the disclosure.

Referring to FIG. 9, an AR device (e.g., the AR device 610 of FIG. 6)may display a second GUI 920 in an AR in addition to a first GUI 910.The first GUI 910 may correspond to the first GUI 30 of FIG. 1 or thefirst GUI 820 of FIG. 8A. The second GUI 920 may indicate a messagethread of a message application. The second GUI 920 may be referred toas a ‘nest’. In this case, the first GUI 910 and the second GUI 920 maybe referred to as a ‘main nest’ and a ‘secondary nest’, respectively.

According to an embodiment of the disclosure, the second GUI 920 mayinclude a first avatar 922 corresponding to a user account of an ARsystem. It may be described that the second GUI 920 ‘includes’ the firstavatar 922. However, according to various embodiments of the disclosure,the AR device may place the first avatar 922 at a location (e.g., top,bottom, side, or inside) adjacent to the second GUI 920.

According to an embodiment of the disclosure, the AR device may detect auser gesture 955 of moving 950 a second avatar 911 to the second GUI 920among avatars 911 and 912 included in the first GUI 910. For example,the AR device may detect a user gesture 955 of grabbing 950 the secondavatar 911 from the first GUI 910 through a hand pose sensor (e.g., thehand pose sensor 551 in FIG. 5), and dropping the grabbed second avatar911 on the second GUI 920. The AR device may provide a graphic effect ofmoving the second avatar 911 from the first GUI 910 to the second GUI920 depending on the detected user gesture 955 in the AR. When thesecond avatar 911 is located on the second GUI 920, a user deviceelectrically connected to the AR device may communicate with acounterpart of a contact corresponding to the second avatar 911.

According to an embodiment of the disclosure, the second GUI 920 may bedisplayed before the second avatar 911 is selected, the second avatar911 may be displayed in response to detecting a user gesture 955 thatdeviates from the first GUI 910.

According to an embodiment of the disclosure, to provide a sense ofreality, such as a talking between the first avatar 922 and the secondavatar 911, an AR device may change locations and directions of thefirst avatar 922 and the second avatar 911. For example, the AR devicemay be arranged such that the first avatar 922 and the second avatar 911stand while facing each other on opposite sides of the second GUI 920.

According to an embodiment of the disclosure, the AR device may displayicons 925 and 926 of menus for controlling a message thread indicated bythe second GUI 920 at a location adjacent to the second GUI 920. Forexample, the menu corresponding to the icons 925 and 926 may include atleast one of a function to terminate the message thread, a function toshare content (e.g., at least one of an image, a video, a link, a map,another person's contact, a music, a file, a voice message, or aschedule), or a function to send an emoticon.

According to an embodiment of the disclosure, the AR device mayterminate communication with a counterpart of a contact indicated by thesecond avatar 911, in response to detecting a user gesture 955 of moving950 the second avatar 911 from the second GUI 920 to the first GUI 910.

FIGS. 10A and 10B illustrate an operation of detecting a gesture for anavatar according to various embodiments of the disclosure. FIGS. 10A and10B illustrate an operation of detecting a movement of a user's finger(or a hand) by using a hand pose sensor.

Referring to FIGS. 10A and 10B, in operation 1001, an AR device maydetect that a user's finger 1050 moves toward an avatar 1012 disposed ona first GUI 1010 (e.g., the first GUI 910 in FIG. 9). The AR device maynot only detect the movement of the user's finger 1050, but may alsomeasure a distance 1015 between the user's finger 1050 and the avatar1012.

When the distance 1015 between the user's finger 1050 and the avatar1012 is less than a threshold value, the AR device may determine thatthe avatar 1012 is selected by the user. According to an embodiment ofthe disclosure, to notify the user that the distance 1015 between theuser's finger 1050 and the avatar 1012 is getting closer, the AR devicemay provide a graphic effect indicating a gesture of the avatar 1012.For example, referring to operation 1004 of FIG. 10B, the AR device mayprovide a graphic effect in which the avatar 1012 raises a hand in adirection, in which the user's finger 1050 is located, in response todetecting that the distance 1065 between the user's finger 1050 and theavatar 1012 is less than the threshold value. According to an embodimentof the disclosure, to guide a location where the user's finger 1050 iscapable of grabbing the avatar 1012, the AR device may output anadditional graphic effect (e.g., light) on a hand raised by the avatar1012. For another example, referring to FIG. 10B, in operation 1005, theAR device may provide a visual graphic effect 1080 (e.g., firecrackersor pollen) at a periphery of the avatar 1012 in response to an eventthat the user's finger 1050 touches a hand of the avatar 1012.

In operation 1002, the AR device may detect a gesture in which theuser's finger 1050 grabs the avatar 1012. For example, when the user'sfinger 1050 touches the avatar 1012 or the distance between the user'sfinger 1050 and the avatar 1012 is less than the threshold value, the ARdevice may detect a movement in which the user's finger 1050 is closed.The AR device may provide a graphic effect in which the avatar 1012deviates from the first GUI 1010, in response to detecting a gesture inwhich the user's finger 1050 grabs the avatar 1012. The AR device mayprovide a graphic effect in which the avatar 1012 moves together withthe user's finger 1050 depending on the movement of the closed user'sfinger 1050.

In operation 1003, the AR device may display the avatar 1012 on a secondGUI 1020 in response to detecting that the user's finger 1050 is openedor detecting that the avatar 1012 is located at a location adjacent tothe second GUI 1020 (e.g., the second GUI 920 in FIG. 9). The second GUI1020 may include an avatar 1022 (e.g., the first avatar 922 of FIG. 9)corresponding to a user account of the AR device.

FIG. 10A illustrates an embodiment in which the avatar 1012 moves fromthe first GUI 1010 to the second GUI 1020. However, the same principlemay be applied to an embodiment in which the avatar 1012 moves from thesecond GUI 1020 to the first GUI 1010.

FIG. 11 is an operation flowchart 1100 of an AR system for providing agraphic effect in which an avatar moves according to an embodiment ofthe disclosure.

Referring to FIG. 11, in operation 1110, an AR system may display asecond GUI (e.g., the second GUI 920 of FIG. 9) indicating a messagethread in an AR. According to an embodiment of the disclosure, thesecond GUI may include an avatar (e.g., the first avatar 922 of FIG. 9)corresponding to a user account of the AR system.

In operation 1120, the AR system may detect a first gesture for anavatar (e.g., the second avatar 911 of FIG. 9) included in the first GUI(e.g., the first GUI 910 of FIG. 9). For example, the AR system maydetect a movement, in which a user's finger moves to a location close toan avatar and the user's finger is closed toward the avatar, using ahand pose sensor. The AR system may detect a movement in which theuser's finger moves from the first GUI to the second GUI while theuser's finger is closed.

In operation 1130, the AR system may provide a graphic effect, in whichthe avatar moves from the first GUI to the second GUI, based on thedetected first gesture.

According to another embodiment of the disclosure, the AR system maydisplay the second GUI in the AR in response to detecting the firstgesture in operation 1120. For example, when it is detected that theuser's finger grabs the avatar, the AR system may display the secondGUI.

FIG. 12 illustrates an operation of changing a message thread accordingto an embodiment of the disclosure.

Referring to FIG. 12, a user device may generate a plurality of messagethreads for counterparts of a plurality of contacts. An AR device maydisplay a plurality of GUIs 1231, 1232, 1233, and 1234 indicating theplurality of message threads in an AR. For example, the AR device maydisplay the plurality of GUIs 1231, 1232, 1233, and 1234 indicating theplurality of message threads at a location adjacent to a second GUI 1220(e.g., the second GUI 920 of FIG. 9) generated between a first avatar1221 and a second avatar 1222.

According to an embodiment of the disclosure, the AR device may select acurrent message thread among the plurality of message threads dependingon a user input. For example, the AR device may select the currentmessage thread among the plurality of message threads depending on amovement in which a user finger 1250 scrolls the plurality of GUIs 1231,1232, 1233, and 1234 in a specified direction (e.g., 1255) or touchesone of the plurality of GUIs 1231, 1232, 1233, and 1234.

FIGS. 13A and 13B illustrate a user interface for entering a messageaccording to various embodiments of the disclosure.

Referring to FIG. 13A, an AR device may display a second GUI 1320 (e.g.,the second GUI 920 of FIG. 9) including a first avatar 1321 (e.g., thefirst avatar 922 in FIG. 9) and a second avatar 1322 (e.g., the secondavatar 911 in FIG. 9). The AR device may display, at a location adjacentto the second GUI 1320, icons 1331, 1332, and 1333 for selecting thetype of a user input to be transmitted to a counterpart of a contactcorresponding to the second avatar 1322 and an icon 1334 fortransmitting a obtained user input to the counterpart of the contactcorresponding to the second avatar 1322. For example, the type of aninput indicated by the icons 1331, 1332, and 1333 may include a voiceinput, a text input, or a reaction input. For example, the reactioninput may refer to a gesture that appears on the first avatar 1321.

Referring to FIG. 13B, for example, the AR device may receive a user'svoice input 1341. The AR device may obtain the voice input 1341 througha microphone (e.g., the input/output interface 530 of FIG. 5) of the ARdevice. As another example, a user device connected to the AR device mayobtain the voice input 1341 through a microphone (e.g., the input module250 in FIG. 2) of the user device, and may transmit the obtained voiceinput 1341 to the AR device. As another example, the AR device mayobtain a text input through a virtual keyboard 1342 displayed in an AR.In this case, the AR device may obtain a text input on the virtualkeyboard 1342 by detecting a movement of a user's finger using a handpose sensor. As another example, the AR device may obtain a text inputthrough a virtual keyboard output on a touch screen of a user device1343. As another example, although not shown in FIG. 13B, the AR devicemay obtain a text input through an input device (e.g., a Bluetoothkeyboard) separately connected to the user device 1343.

According to an embodiment of the disclosure, the AR device may obtain amessage input through a plurality of input means. For example, when anapplication of a user device is operated in an AR mode, the AR devicemay obtain a message input through the virtual keyboard 1342 displayedin the AR and, at the same time, may obtain a message input through thekeyboard output on a touch screen of the user device 1343. According toanother embodiment of the disclosure, the AR device may set a specificinput means as a default means for obtaining a message input. Accordingto another embodiment of the disclosure, the AR device may obtain amessage input through an input means that is last used.

According to an embodiment of the disclosure, the AR device may transmitthe obtained message (e.g., a voice or a text) indicated by a user inputto a counterpart of a contact corresponding to the second avatar 1322.

According to an embodiment of the disclosure, the AR device may providea sense of reality as if a user of the first avatar 1321 speaks directlyto a user of the second avatar 1322, by displaying a text correspondingto the transmitted message or a speech bubble 1350 including a text onthe first avatar 1321.

FIG. 14 illustrates a UI for displaying a message according to anembodiment of the disclosure.

Referring to FIG. 14, an AR device may display a second GUI 1420 (e.g.,the second GUI 920 of FIG. 9) including a first avatar 1421 (e.g., thefirst avatar 922 in FIG. 9) and a second avatar 1422 (e.g., the secondavatar 911 in FIG. 9). The AR device may display a message received froman electronic device of a counterpart or a message transmitted to theelectronic device of the counterpart in an AR in various shapes.

For example, in operation 1401, the AR device may display a speechbubble 1431 including a text corresponding to a message in the AR. Forexample, the speech bubble 1431 may be displayed on an avatar (e.g., thefirst avatar 1421) corresponding to a subject (e.g., a user of the ARdevice) that sends a message. According to an embodiment of thedisclosure, the AR device may control the speech bubble 1431 todisappear after a specified time.

For another example, in operation 1402, the AR device may display a UI1432 indicating a voice message in the AR. When a user input forselecting the UI 1432 is received, the AR device may output a voicethrough a speaker (e.g., the input/output interface 530 of FIG. 5).

According to an embodiment of the disclosure, the AR device may output amessage as a voice or a text depending on user settings. For example,even though the AR device obtains a voice message or a voice input, theAR device may display a text corresponding to a voice in the AR througha speech-to-text (STT) technology. On the other hand, even though the ARdevice obtains a text message or a text input, the AR device may outputa voice corresponding to a text in the AR through a text-to-speech (TTS)technology.

For another example, in operation 1403, the AR device may display a UI1433 indicating message transmission records in the AR. According to anembodiment of the disclosure, the UI 1433 may be scrolled in a specifieddirection (e.g., upward or downward) based on a user gesture. Accordingto an embodiment of the disclosure, when a specified time elapses or thenumber of texts included in the UI 1433 is not less than a specifiednumber, the AR device may reduce visual clutter by making at least partof texts included in the UI 1433 disappear from the AR.

FIG. 15A illustrates an operation of outputting a message depending on aviewing area according to various embodiments of the disclosure.

Referring to FIG. 15A, when an AR device 1505 (e.g., the AR device 610in FIG. 6) is worn on a part (e.g., a head) of the body of a user 1550,an AR space that the user 1550 is capable of viewing may be limited dueto a physical size limitation of a display (e.g., the display 560 ofFIG. 5) of the AR device 1505. In this specification, an area that theuser 1550 is capable of viewing through the AR device 1505 may bereferred to as a ‘viewing area (e.g., the viewing area 1510)’. Forexample, when a second GUI 1520 (e.g., the second GUI 920 in FIG. 9)including a first avatar 1521 (e.g., the first avatar 922 in FIG. 9) anda second avatar 1522 (e.g., the second avatar 911 in FIG. 9) is fixed toa specified location in a real space, the second GUI 920 may bepositioned within the viewing area 1510 or may deviate from the viewingarea 1510, depending on a movement of the AR device 1505.

According to an embodiment of the disclosure, the AR device 1505 mayoutput a message as a text or a voice based on whether the second GUI1520 deviates from the viewing area 1510. For example, when the secondGUI 1520 is located within the viewing area 1510 as shown in the leftdiagram of FIG. 15A, the AR device 1505 may output a message in a textform (e.g., 1525). As another example, when the second GUI 1520 deviatesfrom the viewing area 151 as shown in the right diagram of FIG. 15A, theAR device 1505 may output a message as a voice.

FIG. 15B illustrates an operation flowchart 1500 of an AR device foroutputting a message depending on a viewing area according to anembodiment of the disclosure.

Referring to FIG. 15B, in operation 1560, the AR device 1505 may detecta movement of the AR device 1505. For example, the AR device 1505 maydetect the movement of the AR device 1505 through a head pose sensor(e.g., the head pose sensor 552 in FIG. 5) or a motion sensor.

In operation 1570, the AR device 1505 may determine whether the secondGUI 1520 is located within the viewing area 1510, in response to anevent that the movement is detected.

When the second GUI 1520 is located within the viewing area 1510, inoperation 1580, the AR device 1505 may display a text (e.g., 1525)corresponding to a message through the display.

When the second GUI 1520 is not located within the viewing area 1510, inoperation 1590, the AR device 1505 may output a voice corresponding to amessage through a speaker.

FIG. 16 illustrates a UI for displaying a received message according toan embodiment of the disclosure.

Referring to FIG. 16, after a message thread is generated, a user device(e.g., the user device 620 in FIG. 6) may receive a message from acounterpart of another contact (e.g., a contact corresponding to a thirdavatar 1612) for which the message thread is not generated. To notify auser of a message received from the counterpart for which the messagethread is not generated, an AR device (e.g., the AR device 610 of FIG.6) may provide a graphic effect indicating a gesture of the third avatar1612 included in a first GUI 1610 (e.g., the first GUI 910 of FIG. 9).For example, the AR device may provide a graphic effect indicating agesture in which the third avatar 1612 raises a hand.

FIG. 17 illustrates a second GUI including a plurality of avatars in anAR according to an embodiment of the disclosure.

Referring to FIG. 17, while a second GUI 1720 (e.g., the second GUI 920of FIG. 9) including a first avatar 1722 (e.g., the first avatar 922 inFIG. 9) and a second avatar 1721 (e.g., the second avatar 911 in FIG. 9)is displayed, an AR device (e.g., the AR device 610 of FIG. 6) maydetect a user gesture of moving a third avatar 1712 among avatars 1711and 1712 included in a first GUI 1710 (e.g., the first GUI 910 of FIG.9) to the second GUI 1720. The user gesture of moving the third avatar1712 to the second GUI 1720 may be the same as or similar to the usergesture described with reference to FIG. 9. For example, the AR devicemay detect, through a hand pose sensor (e.g., the hand pose sensor 551of FIG. 5), a movement in which a user's finger 1755 moves the thirdavatar 1712 in a specified direction 1750 while grabbing the thirdavatar 1712 and then drops the third avatar 1712 on the second GUI 1720.

According to an embodiment of the disclosure, a user device connected tothe AR device may simultaneously communicate with a counterpart of thesecond avatar 1721 and a counterpart of the third avatar 1712, inresponse to detecting that the third avatar 1712 is positioned at alocation adjacent to the second GUI 1720. FIG. 17 illustrates anembodiment in which only the third avatar 1712 is added to the secondGUI 1720, but the number of avatars added to the second GUI 1720 is notlimited thereto.

According to an embodiment of the disclosure, as the third avatar 1712is disposed at a location adjacent to the second GUI 1720, the AR devicemay provide a graphic effect such that the avatars 1712, 1721, and 1722included in the second GUI 1720 express greeting each other.

FIG. 18 illustrates an operation of selecting one of a plurality ofavatars disposed on a second GUI according to an embodiment of thedisclosure.

Referring to FIG. 18, an AR device 1805 (e.g., the AR device 610 in FIG.6) may select one of remaining avatars 1822 and 1823 disposed on asecond GUI 1820 other than a first avatar 1821 corresponding to a useraccount of the AR device 1805 in an AR 1800, based on a gaze directionof a user 1810 wearing the AR device 1805. For example, when a messageinput is obtained while a gaze direction 1815 of the user 1810 points tothe second avatar 1822, it may be indicated that the obtained messageinput is used for a counterpart of a contact corresponding to the secondavatar 1822.

Although not illustrated in FIG. 18, according to other embodiments ofthe disclosure, the AR device 1805 may select the second avatar 1822among the avatars 1822 and 1823, in response to obtaining a user voicefor calling an identifier or name of the second avatar 1822, ordetecting that a finger of the user 1810 points to the second avatar1822.

FIG. 19 illustrates an operation of sharing content in an AR accordingto an embodiment of the disclosure.

Referring to FIG. 19, while a second GUI 1920 (e.g., the second GUI 1820in FIG. 18) including a first avatar 1921 and a second avatar 1922 isdisplayed, an AR system may obtain a user input for sharing content inan AR. For example, the content may include at least one of an image, avideo, a link, a map, another person's contact, music, a file, a voicemessage, or a schedule.

For example, in operation 1901, the AR system may detect a user gestureof sharing content 1910 output in the AR. For example, the user gesturemay include a gesture of grabbing the content 1910 and dropping thecontent 1910 on the second GUI 1920 similar to the principle describedwith reference to FIG. 10A.

As another example, in operation 1902, the AR system may obtain a userinput for sharing content 1935 output on a touch screen of a user device1930 (e.g., the user device 620 in FIG. 6) in the AR. For example, theuser input may include a swipe input facing in a specific direction(e.g., upward) on the touch screen.

FIG. 20 illustrates an interaction scenario in a second GUI according toan embodiment of the disclosure.

Referring to FIG. 20, it is assumed that a second GUI 2010 including afirst avatar 2011 and a second avatar 2012 is displayed in an AR.Operation 2001 to operation 2003 are not implemented in order and may beperformed independently.

Referring to FIG. 20, in operation 2001, an AR device (e.g., the ARdevice 610 in FIG. 6) may express a user's emotion through a firstavatar 2011 corresponding to a user account. For example, the AR devicemay display a UI 2015 for selecting various gestures of the first avatar2011, and may control the first avatar 2011 to express a gestureselected in the UI 2015.

In operation 2002, the AR device may display a UI 2025 indicating that avoice message is being output while the voice message received from acounterpart of a contact corresponding to the second avatar 2012 isbeing output.

In operation 2003, the AR device may display an input window 2035 in theAR such that a user is capable of identifying an obtained message input.

FIG. 21 illustrates an image including avatars according to anembodiment of the disclosure.

Referring to FIG. 21, an AR device may provide an image 2110 of avatars2111, 2112, 2113, or 2114 included in a second GUI (e.g., the second GUI2010 of FIG. 20). The number of avatars included in the image 2110 isnot limited to the example illustrated in FIG. 21. For example, the ARdevice may generate the image 2110 in response to a user input forrequesting image capture.

As described above, according to various embodiments of the disclosure,an AR device (e.g., the AR device 501 of FIG. 5) supporting an AR mayinclude a display (e.g., the display 560 of FIG. 5), a communicationcircuit (e.g., the communication interface 540 of FIG. 5), a processor(e.g., the processor 570 of FIG. 5) operatively connected to the displayand the communication circuit, and a memory (e.g., the memory 520 ofFIG. 5) operatively connected to the processor. The memory may storeinstructions that, when executed, cause the processor to establish aconnection with a user device (e.g., the user device 301 of FIG. 3)storing a contact application and a message application, through thecommunication circuit, to detect that the message application isexecuted, and to display a first GUI (e.g., 30 of FIG. 1) and at leastone avatar (e.g., 32, 34, or 36 of FIG. 1), which is disposed at alocation adjacent to the first GUI and corresponds to at least onecontact associated with the contact application or the messageapplication, through the display in the AR.

According to an embodiment of the disclosure, the instructions may causethe processor to display a second GUI (e.g., the second GUI 920 of FIG.9), which indicates a message thread of the message application and isdisposed at a location spaced from the first GUI, and a first avatar(e.g., the first avatar 922 of FIG. 9), which is disposed at a locationadjacent to the second GUI and which corresponds to a user account,through the display in the AR.

According to an embodiment of the disclosure, the AR device may furtherinclude at least one sensor (e.g., the sensor 550 of FIG. 5), amicrophone (e.g., the input/output interface 530 of FIG. 5), and aspeaker (e.g., the input/output interface 530 of FIG. 5). Theinstructions may cause the processor to detect a first gesture for asecond avatar (e.g., the second avatar 911 of FIG. 9) disposed at thelocation adjacent to the first GUI, through the at least one sensor, andto provide a graphic effect in which the second avatar moves from thefirst GUI to the second GUI, through the display in response todetecting the first gesture.

According to an embodiment of the disclosure, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions may cause the processor to receive a first userinput for inputting a message by detecting a second gesture for avirtual keyboard (e.g., the virtual keyboard 1342 of FIG. 13B) displayedthrough the display through the at least one sensor, obtaining a uservoice (e.g., the voice input 1341 of FIG. 13B) through the microphone,or receiving information from the user device or an accessory devicethrough the communication circuit.

According to an embodiment of the disclosure, the instructions may causethe processor to display the message corresponding to the first userinput on the first avatar through the display.

According to an embodiment of the disclosure, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions may cause the processor to receive a first messageof the second avatar through the communication circuit and to output avoice corresponding to the first message through the speaker or todisplay a text including the first message on the second avatar throughthe display.

According to an embodiment of the disclosure, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions may cause the processor to detect a movement ofthe AR device through the at least one sensor, to determine whether thesecond GUI is positioned within a viewing area (e.g., the viewing area1510 of FIG. 15A) of the display, in response to the detected movement,to display the text corresponding to the first message on the secondavatar through the display based on the second GUI being positionedwithin the viewing area, or to output the voice corresponding to thefirst message through the speaker based on the second GUI being notpositioned within the viewing area.

According to an embodiment of the disclosure, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions may cause the processor to detect a third gesturefor a third avatar (e.g., the third avatar 1712 of FIG. 17) disposed atthe location adjacent to the first GUI, through the at least one sensorand to provide a graphic effect in which the third avatar moves from thefirst GUI to the second GUI, in response to detecting the third gesture.

According to an embodiment of the disclosure, while the first avatar,the second avatar, and the third avatar are displayed at the locationadjacent to the second GUI, the instructions may cause the processor toreceive a second user input for selecting one of the second avatar andthe third avatar by detecting a user gaze through the at least onesensor, obtaining a user voice through the microphone, or detecting agesture through the at least one sensor.

According to an embodiment of the disclosure, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions may cause the processor to receive a secondmessage of a fourth avatar (e.g., the third avatar 1612 of FIG. 16)disposed at the location adjacent to the first GUI, through thecommunication circuit and to provide a graphic effect indicating amovement of the fourth avatar through the display in response toreceiving the second message.

According to an embodiment of the disclosure, while the messageapplication is executed, the instructions may cause the processor toblock execution of another application stored in the memory or toexecute the other application in a background.

As described above, according to various embodiments of the disclosure,an AR system supporting an AR may include an AR device (e.g., the ARdevice 501 of FIG. 5) and a user device (e.g., the user device 301 ofFIG. 3). The AR device may include a first display (e.g., the display560 of FIG. 5) and a first communication circuit (e.g., thecommunication interface 540 of FIG. 5). The user device may include asecond display (e.g., the touch screen 320 of FIG. 3), a secondcommunication circuit (e.g., the communication circuit 350 of FIG. 3), aprocessor (e.g., the controller 330 of FIG. 3) operatively connected tothe second display and the second communication circuit, and a memory(e.g., the memory 340 of FIG. 3) operatively connected to the processor.The memory may store at least one contact associated with a messageapplication or a contact application. The memory may store instructionsthat, when executed, cause the processor to establish a connection withthe AR device through the second communication circuit, to detect thatthe message application is executed, to render a first GUI (e.g., 30 ofFIG. 1) and at least one avatar (e.g., 32, 34, or 36 of FIG. 1), whichare to be displayed in the AR through the first display, as a 3dimensional (3D) image, and to transmit the rendered first GUI and therendered at least one avatar to the AR device through the secondcommunication circuit. The at least one avatar may correspond to the atleast one contact.

According to an embodiment of the disclosure, the instructions may causethe processor to render a second GUI (e.g., the second GUI 920 of FIG.9) and a first avatar (e.g., the first avatar 922 of FIG. 9), which areto be displayed in the AR through the first display, as the 3D image andto transmit the rendered second GUI and the rendered first avatar to theAR device through the second communication circuit. The second GUI mayindicate a message thread of the message application and is disposed ata location spaced from the first GUI. The first avatar may be disposedat a location adjacent to the second GUI and corresponds to a useraccount.

According to an embodiment of the disclosure, the AR system may furtherinclude at least one sensor (e.g., the sensor 550 of FIG. 5) detecting agesture of a user, a microphone (e.g., the input module 250 of FIG. 2 orthe input/output interface 530 of FIG. 5), and a speaker (e.g., thesound output module 255 of FIG. 2 or the input/output interface 530 ofFIG. 5). The instructions may cause the processor to detect a firstgesture for a second avatar (e.g., the second avatar 911 of FIG. 9)disposed at a location adjacent to the first GUI, through the at leastone sensor, to render a first graphic effect in which the second avatarmoves from the first GUI to the second GUI, in response to detecting thefirst gesture, and to transmit the rendered first graphic effect to theAR device through the second communication circuit.

According to an embodiment of the disclosure, the instructions may causethe processor to receive a first user input for inputting a message byreceiving a text input through a virtual keyboard displayed on thesecond display or obtaining a user voice through the microphone.

According to an embodiment of the disclosure, the instructions may causethe processor to render the message corresponding to the first userinput as the 3D image to be displayed in the AR through the firstdisplay and to transmit the rendered message to the AR device throughthe second communication circuit.

According to an embodiment of the disclosure, the instructions may causethe processor to receive a first message through the secondcommunication circuit and to output a voice corresponding to the firstmessage through the speaker or to render a text including the firstmessage so as to be displayed in the AR through the first display.

According to an embodiment of the disclosure, the instructions may causethe processor to block execution of another application stored in thememory or to execute the other application in a background, while themessage application is executed.

According to an embodiment of the disclosure, the instructions may causethe processor to detect a third gesture for a third avatar disposed atthe location adjacent to the first GUI, through the at least one sensor,to render a second graphic effect in which the third avatar moves fromthe first GUI to the second GUI, in response to detecting the thirdgesture, and to transmit the rendered second graphic effect to the ARdevice through the second communication circuit.

According to an embodiment of the disclosure, the instructions may causethe processor to receive a second user input for selecting one of thesecond avatar and the third avatar by detecting a user gaze through theat least one sensor, obtaining a user voice through the microphone, ordetecting a gesture through the at least one sensor.

According to an embodiment of the disclosure, the instructions may causethe processor to receive a second message of a fourth avatar disposed atthe location adjacent to the first GUI, through the second communicationcircuit, to render a third graphic effect indicating a movement of thefourth avatar in response to receiving the second message, and totransmit the rendered third graphic effect to the AR device through thesecond communication circuit.

The electronic device according to various embodiments disclosed in thedisclosure may be various types of devices. The electronic device mayinclude, for example, a portable communication device (e.g., asmartphone), a computer device, a portable multimedia device, a mobilemedical appliance, a camera, a wearable device, or a home appliance. Anelectronic device according to an embodiment of this specification maynot be limited to the above-described electronic devices.

Various embodiments of the disclosure and terms used herein are notintended to limit the technical features described in the disclosure tospecific embodiments of the disclosure, and it should be understood thatthe embodiments and the terms include modification, equivalent, oralternative on the corresponding embodiments described herein. Withregard to description of drawings, similar or related components may bemarked by similar reference marks/numerals. The singular form of thenoun corresponding to an item may include one or more of items, unlessinterpreted otherwise in context. In the disclosure, the expressions “Aor B”, “at least one of A and B”, “at least one of A or B”, “A, B, orC”, “at least one of A, B, and C”, and “at least one of A, B, or C” mayinclude any and all combinations of one or more of the associated listeditems. The terms, such as “first” or “second” may be used to simplydistinguish the corresponding component from the other component, but donot limit the corresponding components in other aspects (e.g.,importance or order). When a component (e.g., a first component) isreferred to as being “coupled with/to” or “connected to” anothercomponent (e.g., a second component) with or without the term of“operatively” or “communicatively”, it may mean that a component isconnectable to the other component, directly (e.g., by wire),wirelessly, or through the third component.

The term “module” used herein may include a unit, which is implementedwith hardware, software, or firmware, and may be interchangeably usedwith the terms “logic”, “logical block”, “part”, or “circuit”. The“module” may be a minimum unit of an integrated part or may be a minimumunit of the part for performing one or more functions or a part thereof.For example, according to an embodiment of the disclosure, the modulemay be implemented in the form of an application-specific integratedcircuit (ASIC).

Various embodiments of the disclosure may be implemented with software(e.g., the program 240) including one or more instructions stored in astorage medium (e.g., an embedded memory 236 or an external memory 238)readable by a machine (e.g., the electronic device 201). For example,the processor (e.g., the processor 220) of the machine (e.g., theelectronic device 201) may call at least one instruction of the storedone or more instructions from a storage medium and then may execute theat least one instruction. This enables the machine to operate to performat least one function depending on the called at least one instruction.The one or more instructions may include a code generated by a complieror a code executable by an interpreter. The machine-readable storagemedium may be provided in the form of a non-transitory storage medium.Herein, ‘non-transitory’ just means that the storage medium is atangible device and does not include a signal (e.g., electromagneticwaves), and this term does not distinguish between the case where datais semi-permanently stored in the storage medium and the case where thedata is stored temporarily. For example, the ‘non-transitory storagemedium’ may include a buffer in which data is temporarily stored.

According to an embodiment of the disclosure, a method according tovarious embodiments disclosed herein may be provided to be included in acomputer program product. The computer program product may be tradedbetween a seller and a buyer as a product. The computer program productmay be distributed in the form of a machine-readable storage medium(e.g., a compact disc read only memory (CD-ROM)) or may be distributed(e.g., downloaded or uploaded), through an application store (e.g.,PlayStore™), directly between two user devices (e.g., smartphones), oronline. In the case of on-line distribution, at least part of thecomputer program product (e.g., a downloadable app) may be at leasttemporarily stored in the machine-readable storage medium, such as thememory of a manufacturer's server, an application store's server, or arelay server or may be generated temporarily.

According to various embodiments of the disclosure, each component(e.g., a module or a program) of the above-described components mayinclude a single entity or a plurality of entities. According to variousembodiments of the disclosure, one or more the components of theabove-described components or operations may be omitted, or one or moreother components or operations may be added. Alternatively oradditionally, a plurality of components (e.g., a module or a program)may be integrated into one component. In this case, the integratedcomponent may perform one or more functions of each component of theplurality of components in the manner same as or similar to beingperformed by the corresponding component of the plurality of componentsprior to the integration. According to various embodiments of thedisclosure, operations executed by modules, programs, or othercomponents may be executed by a successive method, a parallel method, arepeated method, or a heuristic method. Alternatively, at least one ormore of the operations may be executed in another order or may beomitted, or one or more operations may be added.

According to the embodiments disclosed in this specification, an ARdevice may provide an environment in which a user is capable ofnaturally exchanging a message with a counterpart in an AR.

According to the embodiments disclosed in this specification, the ARdevice may provide an environment in which the user is capable ofutilizing a message application of a user device without restriction ofactions even while the user wears the AR device.

According to the embodiments disclosed in this specification, the ARdevice may display an execution screen of an application stored in theuser device in a three-dimensional space.

Besides, a variety of effects directly or indirectly understood throughthe specification may be provided.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An augmented reality (AR) device that supports anAR, the AR device comprising: a display; a communication circuit; atleast one processor operatively connected to the display and thecommunication circuit; and a memory operatively connected to the atleast one processor, wherein the memory stores instructions that, whenexecuted, cause the at least one processor to: establish a connectionwith a user device storing a contact application and a messageapplication, through the communication circuit, detect that the messageapplication is executed, and display a first graphic user interface(GUI) and at least one avatar, which is disposed at a location adjacentto the first GUI and corresponds to at least one contact associated withthe contact application or the message application, through the displayin the AR.
 2. The AR device of claim 1, wherein the instructions furthercause the at least one processor to: display a second GUI, whichindicates a message thread of the message application and is disposed ata location spaced from the first GUI, and a first avatar, which isdisposed at a location adjacent to the second GUI and which correspondsto a user account, through the display in the AR.
 3. The AR device ofclaim 2, further comprising: at least one sensor; a microphone; and aspeaker, wherein the instructions further cause the at least oneprocessor to: detect a first gesture for a second avatar disposed at thelocation adjacent to the first GUI, through the at least one sensor, andprovide a graphic effect in which the second avatar moves from the firstGUI to the second GUI, through the display in response to detecting thefirst gesture.
 4. The AR device of claim 3, wherein, while the firstavatar and the second avatar are displayed at the location adjacent tothe second GUI, the instructions further cause the at least oneprocessor to: receive a first user input for inputting a message bydetecting a second gesture for a virtual keyboard displayed through thedisplay through the at least one sensor, obtaining a user voice throughthe microphone, or receiving information from the user device or anaccessory device through the communication circuit.
 5. The AR device ofclaim 4, wherein the instructions further cause the at least oneprocessor to: display the message corresponding to the first user inputon the first avatar through the display.
 6. The AR device of claim 3,wherein, while the first avatar and the second avatar are displayed atthe location adjacent to the second GUI, the instructions further causethe at least one processor to: receive a first message of the secondavatar through the communication circuit, and output a voicecorresponding to the first message through the speaker, or display atext including the first message on the second avatar through thedisplay.
 7. The AR device of claim 6, wherein, while the first avatarand the second avatar are displayed at the location adjacent to thesecond GUI, the instructions further cause the at least one processorto: detect a movement of the AR device through the at least one sensor,determine whether the second GUI is positioned within a viewing area ofthe display, in response to the detected movement, display the textcorresponding to the first message on the second avatar through thedisplay, based on the second GUI being positioned within the viewingarea, or output the voice corresponding to the first message through thespeaker, based on the second GUI being not positioned within the viewingarea.
 8. The AR device of claim 3, wherein, while the first avatar andthe second avatar are displayed at the location adjacent to the secondGUI, the instructions further cause the at least one processor to:detect a third gesture for a third avatar disposed at the locationadjacent to the first GUI, through the at least one sensor, and providea graphic effect in which the third avatar moves from the first GUI tothe second GUI, in response to detecting the third gesture.
 9. The ARdevice of claim 8, wherein, while the first avatar, the second avatar,and the third avatar are displayed at the location adjacent to thesecond GUI, the instructions further cause the at least one processorto: receive a second user input for selecting one of the second avataror the third avatar by detecting a user gaze through the at least onesensor, obtaining a user voice through the microphone, or detecting agesture through the at least one sensor.
 10. The AR device of claim 3,wherein, while the first avatar and the second avatar are displayed atthe location adjacent to the second GUI, the instructions further causethe at least one processor to: receive a second message of a fourthavatar disposed at the location adjacent to the first GUI, through thecommunication circuit, and provide a graphic effect indicating amovement of the fourth avatar through the display in response toreceiving the second message.
 11. The AR device of claim 9, wherein theinstructions further cause the at least one processor to: while themessage application is executed, block execution of another applicationstored in the memory or execute the other application in a background.12. An augmented reality (AR) system supporting an AR, the AR systemcomprising: an AR device; and a user device, wherein the AR deviceincludes: a first display, and a first communication circuit, whereinthe user device includes: a second display, a second communicationcircuit, at least one processor operatively connected to the seconddisplay and the second communication circuit, and a memory operativelyconnected to the at least one processor, wherein the memory stores atleast one contact associated with a message application or a contactapplication, wherein the memory stores instructions that, when executed,cause the at least one processor to: establish a connection with the ARdevice through the second communication circuit, detect that the messageapplication is executed, render a first graphic user interface (GUI) andat least one avatar, which are to be displayed in the AR through thefirst display, as a 3 dimensional (3D) image, and transmit the renderedfirst GUI and the rendered at least one avatar to the AR device throughthe second communication circuit, and wherein the at least one avatarcorresponds to the at least one contact.
 13. The AR system of claim 12,wherein the instructions further cause the at least one processor to:render a second GUI and a first avatar, which are to be displayed in theAR through the first display, as a 3D image, and transmit the renderedsecond GUI and the rendered first avatar to the AR device through thesecond communication circuit, wherein the second GUI indicates a messagethread of the message application and is disposed at a location spacedfrom the first GUI, and wherein the first avatar is disposed at alocation adjacent to the second GUI and corresponds to a user account.14. The AR system of claim 13, further comprising: at least one sensorconfigured to detect a gesture of a user; a microphone; and a speaker,wherein the instructions further cause the at least one processor to:detect a first gesture for a second avatar disposed at a locationadjacent to the first GUI, through the at least one sensor, render afirst graphic effect in which the second avatar moves from the first GUIto the second GUI, in response to detecting the first gesture, andtransmit the rendered first graphic effect to the AR device through thesecond communication circuit.
 15. The AR system of claim 14, wherein theinstructions further cause the at least one processor to: receive afirst user input for inputting a message by receiving a text inputthrough a virtual keyboard displayed on the second display or obtaininga user voice through the microphone.
 16. The AR system of claim 15,wherein the instructions further cause the at least one processor to:render the message corresponding to the first user input as the 3D imageto be displayed in the AR through the first display, and transmit therendered message to the AR device through the second communicationcircuit.
 17. The AR system of claim 14, wherein the instructions furthercause the at least one processor to: receive a first message through thesecond communication circuit, and output a voice corresponding to thefirst message through the speaker or render a text including the firstmessage so as to be displayed in the AR through the first display. 18.The AR system of claim 17, wherein the instructions further cause the atleast one processor to: while the message application is executed, blockexecution of another application stored in the memory or execute theother application in a background.
 19. The AR system of claim 14,wherein the instructions further cause the at least one processor to:detect a third gesture for a third avatar disposed at the locationadjacent to the first GUI, through the at least one sensor, render asecond graphic effect in which the third avatar moves from the first GUIto the second GUI, in response to detecting the third gesture, andtransmit the rendered second graphic effect to the AR device through thesecond communication circuit.
 20. The AR system of claim 19, wherein theinstructions further cause the at least one processor to: receive asecond user input for selecting one of the second avatar or the thirdavatar by detecting a user gaze through the at least one sensor,obtaining a user voice through the microphone, or detecting a gesturethrough the at least one sensor.